The present invention relates generally to interface systems for allowing humans to interface naturally with computer systems and simulations, and, more particularly, to interface systems that allow a user to interact with computer simulated environments both visually and through haptic sensations.
Computer systems are used increasingly to provide simulated experiences to users, for purposes such as training, remote control and sensing, and entertainment. These systems typically include a visual display, such as a standard video monitor, through which visual information generated by the simulation is presented to the user. Virtual reality systems may also include more sophisticated visual displays, such as a head-mounted displays, which aid the user in becoming “immersed” into the simulated environment by attempting to remove any visual input to the user that is not being generated by the simulation. To further enhance the user's simulated experience, sound is provided through speakers or headphones that are connected to the computer system and provide aural information that is controlled by the simulation. In addition, interface devices commonly allow users to interact with the simulation through manual commands or gestures made by the user, i.e., the device tracks the user's “kinesthetic” activities. Keyboards, trackballs, mice, joysticks, pedals, steering wheels, and joypads are interface devices that allow human interaction with computer systems.
Typical human interface devices are input only: They track a users physical activities, but do not provide a means of presenting physical information back to the user. For example, a traditional computer joystick will allow a user to manipulate an object within a computer simulated environment. However, if that object encounters a simulated obstruction, the interaction will only be experienced visually (and maybe aurally through sound feedback), not physically. In other words, when a user manipulates a joystick and causes a computer generated object to encounter a computer generated obstruction, the user will not feel the physical sensation of one object hitting another object. Thus, the user is not truly “immersed” in the simulation as the user receives no physical feedback in conjunction with simulated experiences.
This missing feedback modality can be supplied by a force feedback interface device. A force feedback human interface device is a special class of manual numan interface that not only tracks a user's manual gestures but also includes means of presenting physical sensations back to the user. A force feedback device typically includes sensors for tracking a user's motions and actuators for producing physical forces representative of the simulated interactions. Using a force feedback joystick, a user may manipulate a simulated object in a simulated environment such that if that object encounters forces or other simulated phenomena, the actuators in the device are commanded to simulate a sensation associated with the interaction. For example, if a user manipulates a force feedback joystick to control a simulated brick and causes the brick to contact a simulated piece of metal, the computer would generate forces on the joystick so that the user would feel a physical sensation representative of the encounter. Such physical feedback makes the computer-simulated environment significantly more realistic to the user.
One goal in developing realistic computer simulated environments is to allow users to take advantage of their natural manual dexterity and basic physical skills when interacting with computer simulations. When trying to create a simulated environment in which users can make use of their dexterous skills, it is important to establish a meaningful and intuitive correlation between the information displayed visually and the manual information perceived as “feel” through the force feedback interface device. This is particularly important when trying to create a simulation environment for allowing users to engage in computer-simulated “sporting” interactions or similar simulations which are receptive to a wide range of physical skill in the user. In such applications, a meaningful and intuitive correlation between the visual display of sporting events and manual physical interactions required of that sporting event is critical.
For example, in a simulated sporting environment, a user might wield an interface device which represents a paddle or racquet within the simulated environment. The user will manipulate the simulated paddle and interact with other simulated entities such as pucks, balls, walls, barriers, and even additional paddles manipulated by other players/users of the environment. The interaction between the user's paddle and other simulated entities in the environment will be displayed visually as well as physically. When the user moves the simulated paddle, the user's kinesthetic sensation of paddle location must be reasonably correlated to the representation of paddle location as displayed visually. If the user perceives kinesthetically that his hand has moved to a given location but views a non-corresponding change in visual location, the realism will suffer and the user will be unable to take advantage of his or her full dexterous skills. In some cases, an unnatural correlation between visual and physical experiences will make it impossible for the user to execute the simulated sporting task.
When there are force feedback sensations provided to the user, this correlation between visual and physical becomes even more important. If the user moves a paddle and feels the sensation of the paddle hitting a simulated puck at a given location, but views the paddle-puck interaction at a different location, the realism will suffer and the user will be unable to take advantage of his/her full dexterous skills. Thus while force feedback is intended to increase the realism of a computer simulated environment and enable dexterous manual activities, if done incorrectly, force feedback can disrupt, confuse, and even inhibit a users ability to take advantage of his or her natural dexterous skills.
What is needed therefore is a computer system providing both visual display and force feedback interfacing mechanisms that can establish a natural and meaningful correlation between information displayed visually and physical interactions perceived manually. Unfortunately, there are limitations to force feedback interface devices which make it difficult to represent many simulated physical interactions. For example, a force feedback device has cost, size, and safety constraints which limit the maximum force output that can be applied to a user and therefore make it infeasible to generate sensations corresponding to general interactions between rigid surfaces. Thus, a user may encounter a simulated hard surface, but the user will be able to easily overpower the resistance because of such force output magnitude limitations. However, it is very easy to visually display a depiction of interactions between rigid surfaces which represents a rigid and impenetrable barrier. This dichotomy between the limitations of visual display and physical display must be resolved, especially in simulated sporting interactions where physical skill is central to the simulation. Therefore, there is needed methods for allowing visual display of simulated interactions and physical display of simulated interactions to deviate from their natural mapping at instances when the force feedback device is simply incapable of representing physical interactions which can be represented visually. Such methods must be developed so as not to greatly disrupt a users ability to use his/her natural manual dexterity.